Multicategory Crowdsourcing Accounting for Plurality in Worker Skill and Intention, Task Difficulty, and Task Heterogeneity

摘要

Crowdsourcing allows to instantly recruit workers on the web to annotateimage, web page, or document databases. However, worker unreliability preventstaking a workers responses at face value. Thus, responses from multiple workersare typically aggregated to more reliably infer ground-truth answers. We studytwo approaches for crowd aggregation on multicategory answer spaces stochasticmodeling based and deterministic objective function based. Our stochastic modelfor answer generation plausibly captures the interplay between worker skills,intentions, and task difficulties and allows us to model a broad range ofworker types. Our deterministic objective based approach does not assume amodel for worker response generation. Instead, it aims to maximize the averageaggregate confidence of weighted plurality crowd decision making. In bothapproaches, we explicitly model the skill and intention of individual workers,which is exploited for improved crowd aggregation. Our methods are applicablein both unsupervised and semisupervised settings, and also when the batch oftasks is heterogeneous. As observed experimentally, the proposed methods candefeat tyranny of the masses, they are especially advantageous when there is aminority of skilled workers amongst a large crowd of unskilled and maliciousworkers.